Performance Assessment by Resampling: Rigid Motion Estimators

Quantitative assessment of performance in image understanding tasks with real data is di cult since the data is complex and the di erent computational modules most often interact. Employing modern statistical techniques we have developed a set of numerical tools which provide rigorous performance measures derived solely from the given input. Covariance matrices and con dence intervals are computed for the estimated parameters and individually for the corrected data points. As an example, the proposed methodology is applied to compare rigid motion estimators. 1: Performance assessment in image understanding The lack of universally accepted, rigorous performance assessment methodology is considered by many as one of the major bottlenecks of progress in image understanding. In a recent paper Christensen and Forstner [7] discuss several objections against the widespread use of evaluation techniques. Most of these objections are well justi ed. Performance for real data often cannot be reliably predicted from the controlled experiments with simple, synthetic inputs. A practical system contains several interacting computational modules, and traditional techniques (like error propagation) may be unfeasible. Empirical testing often yields incompatible performance measures and thus ignores the very goal of the procedure, enabling comparison with other techniques. The number of experiments required to obtain highly accurate statistical measures is prohibitively large [14], and for real data it is not well understood how the input space can be parameterized. In this paper we present a new, very general performance assessment methodology which hopefully provides at least partial response to these objections. The new method is an extension of the resampling paradigm widely used in statistics, and has a solid theoretical basis. The method is entirely numerical and derives statistically valid measures solely from the given input. To obtain similar measures using more traditional methods a large set of distinct inputs has to be available. Being data driven the new method belongs to the class of empirical techniques. It retains the advantage of the empirical techniques (they are useful in real situations), but eliminates their often ad-hoc nature. When applied in analytically tractable cases they will reproduce (within the computational accuracy) the theoretically predicted results. However, the often considerable mathematical sophistication needed to obtain these results is no longer necessary. Being a numerical method, it can be applied to any complex system as long as the underlying mild assumptions about the measurement errors are satis ed. The rst use of the resampling paradigm in a low-level image understanding problem (edge detection) is given in [6]. Here we introduce a more general approach which can be applied to performance assessment in most vision tasks. In Section 2 a short review of the employed resampling methods is provided. The proposed performance assessment tools are developed in Section 3. In Section 4 the problem of rigid motion estimation is analyzed and three di erent algorithms are discussed. The performance of these three algorithms is compared in Section 5. 2: Review of resampling techniques Let the data set contain n independent, identically distributed (i.i.d.) measurements zi 2 R , Z = fz1; z2; ;zng, with an unknown probability distribution F . The distribution is partially characterized by the value of some parameter of interest = t(F ) 2 R which is estimated from the data by the estimator ̂ = S [Z]. Resampling techniques are numerical methods to determine from the available single data set statistical measures (like bias or variance) of ̂, usually without any assumption about F . It is important to notice that the data in image understanding may not satisfy the i.i.d. condition. For example, in conic tting and tasks involving the epipolar geometry, the estimate of a bilinear form is computed from non-i.i.d. data, and the quasi-optimal estimators must take this into account [21]. The application of resampling methods to non-i.i.d. data can lead to inconsistent results unless additional safeguards are included. In the following we review only material relevant to the topic of the paper. 2.1: Bootstrap The bootstrap methodology was proposed by Bradley Efron in 1979. An excellent review of bootstrap can be found in [10] and additional material in [2][23][32]. Let F̂n be the empirical distribution function of Z, i.e. obtained by associating equal probability 1 n with each measurement zi. New data sets Z b = fz b 1 ;z b 2 ; ;z b n g called bootstrap samples are formed by sampling with replacement from Z. For each bootstrap sample Z , b = 1; 2; : : : ; B, the statistics ̂ b = S h Z b i is computed. The ensemble of ̂ b is used instead of the sampling distribution of ̂ and the accuracy of ̂ as an estimator of is inferred from this ensemble. In a bootstrap sample Z b each zi appears ki times, such that Pn i=1 ki = n. Thus the bootstrap samples can also be generated by keeping Z xed and varying the probability of each component zi. Therefore, Z b is characterized by the vector p b = h k1 n k2 n kn n i > , with np b obtained by sampling from a multinomial distribution with n draws and equal class probabilities [10, pp. 285{287]. The bootstrap statistics ̂ b can be expressed either as a function of the probability vector ̂ b = t F̂ b n = T h p b i , or as a function of the input data ̂ b = S h Z b i . Similarly, ̂ can be written as ̂ = t F̂n = T [p0], where p0 = h 1 n 1 n 1 n i > . Both formulations will be used in the following sections. The bootstrap estimate of the covariance matrix of ̂ is